Papers with subword regularization
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (P18-1)
Copied to clipboard
| Challenge: | Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation. |
| Approach: | They propose a method to regularize subword segmentations probabilistically by sampling subwords . they also propose 'unigram' language model to be used for better subword sampling . |
| Outcome: | The proposed method improves on low resource and out-of-domain settings with multiple corpora. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to reduce the adverse effect of annotation errors are time-consuming because they require many trained models to detect errors. |
| Approach: | They propose a method that uses a tokenization technique called subword regularization to simulate multiple error detection models for detecting errors. |
| Outcome: | The proposed method performs weighting weighting four to five times faster than existing methods and improves in document classification and named entity recognition tasks. |
BPE-Dropout: Simple and Effective Subword Regularization (2020.acl-main)
Copied to clipboard
| Challenge: | Subword segmentation is widely used to address the open vocabulary problem in machine translation. |
| Approach: | They propose a method that stochastically corrupts the segmentation procedure of BPE and produces multiple segmentations within the same fixed BPE framework. |
| Outcome: | The proposed method produces multiple segmentations within the same fixed BPE framework. |
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing subword regularizations use multiple segmentations during training but only use one segmentation in inference. |
| Approach: | They propose an inference strategy that uses multiple subword segmentations to solve this discrepancy in the training process and inference. |
| Outcome: | The proposed strategy reduces the cost of training and improves the performance of models trained with subword regularization in low-resource machine translation tasks. |
Word-level Perturbation Considering Word Length and Compositional Subwords (2022.findings-acl)
Copied to clipboard
| Challenge: | Word replacement considering length and compositional word replacement are effective word-level perturbations. |
| Approach: | They propose two simple modifications for word-level perturbation: Word Replacement considering Length and Compositional Word Replacement. |
| Outcome: | The proposed methods improve word-level perturbation and classification performance. |
MaxMatch-Dropout: Subword Regularization for WordPiece (2022.coling-1)
Copied to clipboard
| Challenge: | Existing subword regularization methods are specialized to a particular tokenizer type. |
| Approach: | They propose a subword regularization method for WordPiece that uses a maximum matching algorithm for tokenization. |
| Outcome: | The proposed method improves the performance of text classification and machine translation tasks as well as other subword regularization methods. |
Distributional Properties of Subword Regularization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training. |
| Approach: | They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word. |
| Outcome: | The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus. |
Evaluating Robustness to Input Perturbations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that Neural Machine Translation models are brittle to small perturbations in the input. |
| Approach: | They propose to use subword regularization to measure the relative degradation and changes in translation when perturbations are added to the input. |
| Outcome: | The proposed measures show that the models are more robust to perturbations when subword regularization methods are used. |